# MCP Crypto Data Server
A production-grade Python FastAPI server that fetches, normalizes, serves, and streams real-time and historical cryptocurrency market data from major exchanges using CCXT and CoinMarketCap as primary data sources.
## Table of Contents
- [Features](#features)
- [Architecture](#architecture)
- [Tech Stack](#tech-stack)
- [Quick Start](#quick-start)
- [API Documentation](#api-documentation)
- [Configuration](#configuration)
- [Development](#development)
- [Testing](#testing)
- [Docker Deployment](#docker-deployment)
- [Rate Limiting & Best Practices](#rate-limiting--best-practices)
- [Extending the Server](#extending-the-server)
- [Limitations](#limitations)
- [License](#license)
## Features
✅ **REST API Endpoints**
- Ticker data (current prices)
- OHLCV historical candles
- Orderbook snapshots
- Recent trades
✅ **Real-Time Streaming**
- Server-Sent Events (SSE) for ticker streams
- WebSocket support for live updates
- Configurable price change thresholds
- Heartbeat events for connection health
✅ **Multi-Exchange Support**
- Binance, Coinbase Pro, Kraken (via CCXT)
- Extensible to 100+ exchanges
- Unified data normalization across exchanges
✅ **Robust Error Handling**
- Exponential backoff with jitter
- Rate limit detection and handling
- Graceful fallbacks
- Comprehensive error responses
✅ **Caching & Performance**
- Redis integration with in-memory fallback
- Configurable TTL per endpoint
- Reduced API calls and improved response times
✅ **Production Ready**
- Full type annotations with Pydantic
- Structured JSON logging
- Docker & Docker Compose support
- GitHub Actions CI/CD
- High test coverage (≥90%)
## Architecture
### System Design
```
┌─────────────────────────────────────────────────────────────┐
│ FastAPI Application │
├─────────────────────────────────────────────────────────────┤
│ ┌──────────────────────────────────────────────────────┐ │
│ │ API Endpoints (v1) │ │
│ │ ├─ GET /v1/health │ │
│ │ ├─ GET /v1/markets/ticker │ │
│ │ ├─ GET /v1/markets/ohlcv │ │
│ │ ├─ GET /v1/markets/orderbook │ │
│ │ ├─ GET /v1/markets/trades │ │
│ │ ├─ GET /v1/stream/ticker (SSE) │ │
│ │ └─ WS /ws/ticker (WebSocket) │ │
│ └──────────────────────────────────────────────────────┘ │
│ ┌──────────────────────────────────────────────────────┐ │
│ │ Services Layer │ │
│ │ ├─ CCXT Client (Exchange API wrapper) │ │
│ │ ├─ CMC Client (CoinMarketCap fallback) │ │
│ │ ├─ Data Normalizer (canonical models) │ │
│ │ └─ Cache Manager (Redis + in-memory) │ │
│ └──────────────────────────────────────────────────────┘ │
│ ┌──────────────────────────────────────────────────────┐ │
│ │ Utilities │ │
│ │ ├─ Backoff & Retry logic │ │
│ │ ├─ Error handling │ │
│ │ └─ Logging configuration │ │
│ └──────────────────────────────────────────────────────┘ │
└─────────────────────────────────────────────────────────────┘
↓ ↓ ↓
┌─────────┐ ┌─────────┐ ┌──────────┐
│ CCXT │ │ CMC │ │ Redis │
│Exchange │ │ API │ │ Cache │
│ APIs │ │ │ │ │
└─────────┘ └─────────┘ └──────────┘
```
### Key Design Decisions
**Async-First Architecture**: The entire stack uses async/await for non-blocking I/O, enabling high concurrency and efficient resource utilization.
**Normalization Layer**: Exchange-specific responses are normalized into canonical Pydantic models, providing a unified interface regardless of the underlying exchange.
**Caching Strategy**: Multi-tier caching with Redis as primary and in-memory fallback ensures high performance and graceful degradation.
**Rate Limiting**: Per-exchange rate tracking with exponential backoff prevents API bans and ensures sustainable usage.
**Error Handling**: Custom exceptions map to appropriate HTTP status codes (400, 429, 502, 503) with helpful error messages.
## Tech Stack
| Component | Technology | Version |
|-----------|-----------|---------|
| **Runtime** | Python | 3.11+ |
| **Web Framework** | FastAPI | 0.104+ |
| **ASGI Server** | Uvicorn | 0.24+ |
| **Exchange API** | CCXT | 4.0+ |
| **HTTP Client** | httpx | 0.25+ |
| **Caching** | Redis / aioredis | 5.0+ / 2.0+ |
| **Data Validation** | Pydantic | 2.0+ |
| **Testing** | pytest / pytest-asyncio | 7.4+ / 0.21+ |
| **Linting** | ruff | 0.1+ |
| **Containerization** | Docker | Latest |
| **CI/CD** | GitHub Actions | Built-in |
## Quick Start
### Prerequisites
- Python 3.11+
- Docker & Docker Compose (for containerized deployment)
- Git
### Installation
1. **Clone the repository**
```bash
git clone https://github.com/yourusername/mcp-crypto-data-server.git
cd mcp-crypto-data-server
```
2. **Create virtual environment**
```bash
python3.11 -m venv venv
source venv/bin/activate # On Windows: venv\Scripts\activate
```
3. **Install dependencies**
```bash
pip install -e ".[dev]"
```
4. **Setup environment**
```bash
cp .env.example .env
# Edit .env with your configuration
```
5. **Run the server**
```bash
python -m uvicorn app.main:app --reload
```
The server will be available at `http://localhost:8000`
### Docker Deployment
```bash
# Build and run with Docker Compose
cd docker
docker-compose up --build
# Server will be available at http://localhost:8000
# Redis will be available at localhost:6379
```
## API Documentation
### Interactive API Docs
- **Swagger UI**: http://localhost:8000/docs
- **ReDoc**: http://localhost:8000/redoc
- **OpenAPI Schema**: http://localhost:8000/openapi.json
### Endpoints
#### 1. Health Check
```bash
curl -X GET http://localhost:8000/v1/health
```
**Response:**
```json
{
"status": "ok",
"uptime": 123.45,
"version": "0.1.0"
}
```
#### 2. Get Ticker (Current Price)
```bash
curl -X GET "http://localhost:8000/v1/markets/ticker?exchange=binance&symbol=BTC/USDT"
```
**Query Parameters:**
- `exchange` (required): Exchange name (e.g., `binance`, `kraken`, `coinbasepro`)
- `symbol` (required): Trading symbol (e.g., `BTC/USDT`)
- `convert` (optional): Fallback conversion reference
**Response:**
```json
{
"exchange": "binance",
"symbol": "BTC/USDT",
"bid": 42500.0,
"ask": 42510.0,
"last": 42505.0,
"timestamp": 1699000000000,
"raw": {}
}
```
**Caching:** 2 seconds TTL
#### 3. Get OHLCV (Historical Candles)
```bash
curl -X GET "http://localhost:8000/v1/markets/ohlcv?exchange=binance&symbol=BTC/USDT&interval=1h&limit=100"
```
**Query Parameters:**
- `exchange` (required): Exchange name
- `symbol` (required): Trading symbol
- `interval` (optional, default: `1m`): Candle interval (`1m`, `5m`, `15m`, `1h`, `1d`)
- `since` (optional): Fetch data since timestamp (milliseconds)
- `limit` (optional, default: 500, max: 2000): Number of candles
**Response:**
```json
{
"exchange": "binance",
"symbol": "BTC/USDT",
"interval": "1h",
"candles": [
{
"timestamp": 1699000000000,
"open": 42000.0,
"high": 42500.0,
"low": 41900.0,
"close": 42400.0,
"volume": 100.5
}
]
}
```
**Caching:** 5 minutes TTL
#### 4. Get Orderbook
```bash
curl -X GET "http://localhost:8000/v1/markets/orderbook?exchange=binance&symbol=BTC/USDT&limit=50"
```
**Query Parameters:**
- `exchange` (required): Exchange name
- `symbol` (required): Trading symbol
- `limit` (optional, default: 50): Number of levels
**Response:**
```json
{
"exchange": "binance",
"symbol": "BTC/USDT",
"bids": [
{"price": 42500.0, "amount": 1.5},
{"price": 42490.0, "amount": 2.0}
],
"asks": [
{"price": 42510.0, "amount": 1.5},
{"price": 42520.0, "amount": 2.0}
],
"timestamp": 1699000000000
}
```
**Caching:** 1 second TTL
#### 5. Get Recent Trades
```bash
curl -X GET "http://localhost:8000/v1/markets/trades?exchange=binance&symbol=BTC/USDT&limit=100"
```
**Query Parameters:**
- `exchange` (required): Exchange name
- `symbol` (required): Trading symbol
- `limit` (optional, default: 100): Number of trades
**Response:**
```json
{
"exchange": "binance",
"symbol": "BTC/USDT",
"trades": [
{
"trade_id": "12345",
"timestamp": 1699000000000,
"price": 42500.0,
"amount": 0.5,
"side": "buy"
}
]
}
```
**Caching:** 1 second TTL
#### 6. Stream Ticker (SSE)
```bash
curl -X GET "http://localhost:8000/v1/stream/ticker?exchange=binance&symbol=BTC/USDT"
```
**Response:** Server-Sent Events stream
```
data: {"event_type":"ticker","exchange":"binance","symbol":"BTC/USDT","bid":42500.0,"ask":42510.0,"last":42505.0,"timestamp":1699000000000,"price_change_percent":0.5}
data: {"event_type":"heartbeat","timestamp":1699000030000}
```
#### 7. Stream Ticker (WebSocket)
```bash
# Using websocat or similar WebSocket client
websocat ws://localhost:8000/ws/ticker?exchange=binance&symbol=BTC/USDT
```
**Response:** JSON stream of ticker events
```json
{"event_type":"ticker","exchange":"binance","symbol":"BTC/USDT","bid":42500.0,"ask":42510.0,"last":42505.0,"timestamp":1699000000000,"price_change_percent":0.5}
```
## Configuration
### Environment Variables
Create a `.env` file based on `.env.example`:
```bash
# Application
APP_ENV=development # development or production
LOG_LEVEL=INFO # DEBUG, INFO, WARNING, ERROR
PORT=8000 # Server port
HOST=0.0.0.0 # Server host
# Redis
REDIS_URL=redis://localhost:6379/0 # Redis connection URL
REDIS_ENABLED=true # Enable/disable Redis
# CoinMarketCap
CMC_API_KEY= # CoinMarketCap API key (optional)
CMC_API_URL=https://pro-api.coinmarketcap.com/v1
# Exchanges
ENABLED_EXCHANGES=binance,kraken,coinbasepro
# Caching TTLs (seconds)
TICKER_CACHE_TTL=2
ORDERBOOK_CACHE_TTL=1
TRADES_CACHE_TTL=1
OHLCV_CACHE_TTL=300
# Streaming
TICKER_DELTA_THRESHOLD=0.001 # 0.1% price change threshold
STREAM_HEARTBEAT_INTERVAL=30 # Heartbeat interval (seconds)
# Rate Limiting
RATE_LIMIT_REQUESTS=100 # Requests per window
RATE_LIMIT_WINDOW=60 # Window in seconds
# Retry Configuration
MAX_RETRIES=3
INITIAL_BACKOFF=1
MAX_BACKOFF=60
BACKOFF_MULTIPLIER=2
```
## Development
### Project Structure
```
mcp-crypto-data-server/
├── app/
│ ├── __init__.py
│ ├── main.py # FastAPI app factory
│ ├── config.py # Settings configuration
│ ├── api/
│ │ ├── v1.py # Router registration
│ │ └── endpoints/
│ │ ├── health.py # Health check
│ │ ├── market.py # Market endpoints
│ │ └── stream.py # Streaming endpoints
│ ├── services/
│ │ ├── ccxt_client.py # CCXT wrapper
│ │ ├── cmc_client.py # CoinMarketCap client
│ │ ├── cache.py # Caching service
│ │ └── normalizer.py # Data normalization
│ ├── models/
│ │ └── market_models.py # Pydantic models
│ └── utils/
│ ├── errors.py # Custom exceptions
│ ├── backoff.py # Retry/backoff logic
│ └── logging_config.py # Logging setup
├── tests/
│ ├── conftest.py # Pytest fixtures
│ ├── unit/
│ │ ├── test_normalizer.py
│ │ ├── test_cache.py
│ │ └── test_ccxt_client.py
│ └── api/
│ ├── test_endpoints_market.py
│ └── test_streaming.py
├── docker/
│ ├── Dockerfile
│ └── docker-compose.yml
├── .github/
│ └── workflows/
│ └── ci.yml
├── pyproject.toml
├── .env.example
├── README.md
├── LICENSE
└── .gitignore
```
### Running Tests
```bash
# Run all tests
pytest
# Run with coverage
pytest --cov=app --cov-report=html
# Run specific test file
pytest tests/unit/test_normalizer.py
# Run with verbose output
pytest -v
# Run only unit tests
pytest tests/unit/
# Run only integration tests
pytest tests/api/
```
### Code Quality
```bash
# Lint with ruff
ruff check app tests
# Format code
ruff format app tests
# Type checking (optional)
mypy app --ignore-missing-imports
```
## Testing
### Test Coverage
The project aims for ≥90% test coverage on core modules:
- **app/services/**: Normalizer, Cache, CCXT Client
- **app/api/endpoints/**: Market endpoints, Streaming
- **app/utils/**: Error handling, Backoff logic
### Test Categories
**Unit Tests** (`tests/unit/`):
- Data normalization across exchanges
- Cache operations (Redis and in-memory)
- CCXT client error handling
- Backoff and retry logic
**Integration Tests** (`tests/api/`):
- API endpoint responses
- Streaming event formats
- Error handling and status codes
- Caching behavior
### Running Tests with Docker
```bash
docker-compose -f docker/docker-compose.yml exec app pytest
```
## Docker Deployment
### Build Docker Image
```bash
docker build -f docker/Dockerfile -t mcp-crypto-data-server:latest .
```
### Run with Docker Compose
```bash
cd docker
docker-compose up --build
```
### Environment for Docker
Create a `.env` file in the `docker/` directory:
```bash
CMC_API_KEY=your_api_key_here
```
### Health Check
```bash
curl http://localhost:8000/v1/health
```
### Logs
```bash
docker-compose logs -f app
```
### Stop Services
```bash
docker-compose down
```
## Rate Limiting & Best Practices
### Rate Limit Handling
The server implements **per-exchange rate limiting** with exponential backoff:
1. **Rate Limit Detection**: Monitors API responses for rate limit headers
2. **Exponential Backoff**: Automatically backs off with jitter to avoid thundering herd
3. **Graceful Degradation**: Falls back to cached data when rate limited
### Best Practices
**For API Consumers:**
1. **Cache Aggressively**: Respect endpoint TTLs to reduce API calls
- Ticker: 2 seconds
- Orderbook: 1 second
- Trades: 1 second
- OHLCV: 5 minutes
2. **Use Streaming for Real-Time**: SSE/WebSocket for live updates instead of polling
3. **Handle Errors**: Implement retry logic with exponential backoff on 429/503
4. **Monitor Rate Limits**: Check response headers for rate limit information
**For Server Operators:**
1. **Configure Appropriately**: Adjust `RATE_LIMIT_REQUESTS` and `RATE_LIMIT_WINDOW` based on your API keys
2. **Use Redis**: Enable Redis for better cache performance in production
3. **Monitor Logs**: Watch for rate limit errors and adjust configuration
4. **API Keys**: Keep CoinMarketCap API key secure, use environment variables
## Extending the Server
### Adding a New Exchange
1. **Update Configuration**:
```bash
ENABLED_EXCHANGES=binance,kraken,coinbasepro,your_exchange
```
2. **Verify CCXT Support**:
```python
import ccxt
print(ccxt.exchanges) # Check if your exchange is supported
```
3. **Test Normalization**: Add test cases in `tests/unit/test_normalizer.py`
### Adding a New Endpoint
1. **Create endpoint file** in `app/api/endpoints/`
2. **Define Pydantic models** in `app/models/market_models.py`
3. **Implement service logic** in `app/services/`
4. **Register router** in `app/api/v1.py`
5. **Add tests** in `tests/api/`
### Adding Authentication
1. Implement API key validation middleware
2. Add rate limiting per API key
3. Update OpenAPI documentation
## Limitations
### Known Limitations
1. **CoinMarketCap Integration**: Historical OHLCV from CMC requires premium API key. Currently returns mock data with documentation.
2. **Exchange-Specific Features**: Some exchanges have unique features not covered by CCXT's unified API. These are exposed in the `raw` field.
3. **WebSocket Limitations**: WebSocket connections are per-client. For scalability, consider using a message broker (Redis Streams, Kafka).
4. **Real-Time Latency**: Streaming updates depend on exchange API latency and network conditions.
### Future Enhancements
- [ ] Persistent storage for historical candles (SQLite/Timescale)
- [ ] Background job for OHLCV backfilling
- [ ] API key-based rate limiting per consumer
- [ ] Advanced authentication (OAuth2, API keys)
- [ ] Metrics and monitoring (Prometheus)
- [ ] GraphQL API
- [ ] WebSocket multiplexing for better scalability
## License
This project is licensed under the MIT License. See the [LICENSE](LICENSE) file for details.
## Support & Contributing
For issues, questions, or contributions, please open an issue or pull request on GitHub.
### Reporting Issues
Please include:
- Python version
- Operating system
- Steps to reproduce
- Error messages and logs
### Contributing
1. Fork the repository
2. Create a feature branch
3. Make your changes
4. Add tests
5. Submit a pull request
---
**Happy trading! 🚀**